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MULT I -AGENT AUTONOMOUS SYSTEM 

5 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims the benefit of U.S. Provisional 
Patent Application No. 60/398,052, filed July 22, 2002 and 
U.S. Provisional Patent Application No. 60/473,726, filed 
10 May 28, 2003, each of which are hereby incorporated by 
reference as if fully stated herein. 

BACKGROUND OF THE INVENTION 

This invention relates generally to autonomous agents and 
15 more specifically to autonomous agents for exploration of 
hazardous or inaccessible locations. 

Robotic reconnaissance operations are called for in 
potentially hazardous and/or inaccessible situations such as 
remote planetary surfaces. One approach to reconnaissance 

20 

operatxons is to use a few sophisticated, expensive, highly 
capable and reasoning surface-based reconnaissance craft 
(e.g., rovers). In this case the reconnaissance mission is 
lost or adversely affected when one of the few robotic 
reconnaissance craft is damaged or destroyed because there is 
no redundancy . 

In addition, rovers are spatially constrained in that 
they may only view a small portion of an explored area at a 
time. For example, as most rovers are tracked or wheeled 
surface-based craft, their elevation above the ground provides 
only a limited viewing range. Therefore, it is difficult for 
a rover to view a large enough area to make an intelligent 
decision about what features in an operational area are worthy 
of additional investigation. 

The spatial constraint of a rover also causes 

35 

difficulties when planning a path for the rover to follow when 
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traveling through the operational area. As such, the rover 
may construct a locally optimal path through an operational 

5 area but not a globally optimal path because the rover may not 

be able to view a large enough area. 

As an intelligent rover is expensive, both from the 
perspective of the capital cost to build and the resource cost 
to deploy and operate, only a single rover is typically 

10 deployed within an operational area. This means that any 
operational area is constrained to an area no larger than what 
a single rover can explore. 

Finally, because loss of a single rover during an 
exploration mission may be catastrophic from the perspective 

15 of accomplishing an exploration mission's goals, there has 
been a reluctance within the robotics community to allow a 
rover to be truly autonomous . True autonomy means that a 
rover would be able to make its own decisions about where to 
go within an exploration space. If the logic controlling the 

2 0 operations of the rover is faulty, and the rover makes a 

decision that causes it to become inoperable, the mission may 
be lost. As such, most currently deployed rovers are not 
truly autonomous as they are at least partially controlled by 
teleoperation by a human. 
25 Therefore, a need exists for a robotic reconnaissance 

system that uses inexpensive surface-based craft that are 
inexpensive enough, both in terms of capital cost and 
operational resources, that multiple surface-based craft can 
be deployed during a mission. Multiple surface-based craft 

3 0 provide redundancy in the case a surface-based craft is lost. 

In addition, multiple surface-based craft may explore a larger 
area than a single rover. Finally, as loss of one or more of 
the multiple surface-based craft will not destroy a mission, 
the robotic reconnaissance system may be allowed more autonomy 
3 5 in making decisions about what paths to take while exploring 
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an area. Aspects of the present invention meet such need. 

5 SUMMARY OF THE INVENTION 

A multi-agent autonomous system for exploration of 
hazardous or inaccessible locations. The multi-agent 

autonomous system includes simple surface-based agents or 
craft controlled by an airborne tracking and command system. 

10 The airborne tracking and command system includes an 
instrument suite used to image an operational area and any 
craft deployed within the operational area. The image data is 
used to identify the craft, targets for exploration, and 
obstacles in the operational area. The tracking and command 

15 system determines paths for the surface-based craft using the 
identified targets and obstacles and commands the craft using 
simple movement commands to move through the operational area 
to the targets while avoiding the obstacles. Each craft 
includes its own instrument suite to collect surface-based 

20 information about the operational area that is transmitted 
back to the tracking and command system. The tracking and 
command system may be further coupled to a satellite system to 
provide additional image information about the operational 
area. 

25 In one aspect of the invention, a method for controlling 

a surface-based craft within an operational area is provided. 
The method includes providing a tracking and command system 
coupled to the surface-based craft through a transceiver. The 
tracking and command system generates an image of an 

30 operational area and uses the image to generate a path for the 
surface-based craft. The tracking and command system then 
generates a set of craft commands for the surface-based craft 
using the path and transmits the craft commands to the 
surface-based craft via the transceiver. In response to the 

35 craft commands, the surface-based craft moves through the 
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operational area. 

In another aspect of the invention, generating a path for 

5 the surface-based craft further includes identifying the 

surface-based craft's position within the operational area and 
identifying a target by the tracking and command system using 
the image. The tracking and command system then determines a 
path between the craft's position and the target. 

10 In another aspect of the invention, the surface-based 

craft further includes an instrument suite. Generating a path 
for the surface-based craft further includes collecting 
surface-based information from the instrument suite and 
transmitting the surface-based information from the craft to 

15 the tracking and command system. The tracking and command 
system may then generate a path for the surface-based craft 
using the surface-based information. 

In another aspect of the invention, the tracking and 
command system is airborne. The tracking and command system 

20 may be supported by a lighter-than-air or a heavier-than-air 
aircraft. The lighter-than-air aircraft may be tethered or may 
include a thrust generating element. 

In another aspect of the invention, the surface-based 
craft includes a proximity detector and a controller 

25 programmed to use signals from the proximity detector to avoid 
collisions . 

In another aspect of the invention, a multi -agent 
autonomous system includes a tracking and command system 
having a transceiver, an operational area imager, a surface- 
30 based craft path planning module coupled to the operational 
area imager and the transceiver. The system further includes 
a plurality of surface-based craft coupled to the tracking and 
command system through the transceiver. 

In another aspect of the invention, the multi -agent 
35 autonomous system further includes a surface-based craft 
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position module and a reconnaissance target identification 
module coupled to the operational area imager and the path 

5 planning module. 

In another aspect of the invention, the surface-based 
craft further include instrument suites. 

In another aspect of the invention, the tracking and 
command system is airborne. 

10 In another aspect of the invention, the tracking and 

command system includes a processor and a memory coupled to 
the processor. The memory is used to store program 

instructions executable by the processor. The program 
instructions include generating an image of an operational 

15 area; generating a path for the surface-based craft using the 
image; generating a set of craft commands for the surface- 
based craft using the path; and transmitting the craft 
commands to the surface-based craft via a transceiver. 

In another aspect of the invention, the program 

20 instructions for generating a path for the surface-based craft 
further include identifying the surface-based craft's position 
within the operational area using the image, identifying a 
target using the image, and determining a path between the 
craft's position and the target. 

25 In another aspect of the invention, the surface-based 

craft further includes an instrument suite and the program 
instructions for generating a path for the surface-based craft 
further include receiving surface-based information collected 
from the instrument suite by the craft, transmitting the 

3 0 surface-based information from the craft to the tracking and 
command system, and generating a path for the surface-based 
using the surface-based information and the image. 

In another aspect of the invention, the surface-based 
craft further includes a proximity sensor, a drive mechanism, 

35 and a controller coupled to the proximity sensor and drive 
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mechanism. The controller is programmed to avoid collisions 
using signals received from the proximity sensor. 

5 In another aspect of the invention, a multi -agent 

autonomous system includes a self-propelled surface-based 
craft deployed in an operational area and a tracking and 
command system coupled to the plurality of surface-based 
craft . The tracking and command system includes an imager for 

10 generating an image of the operational area coupled to a path 
planer for planning a path for the surface-based craft using 
the image. A craft command generator uses the path to 
generate craft commands for use by a craft commander which 
transmits the craft commands to the surface-based craft 

15 In another aspect of the invention, the multi -agent 

autonomous system further includes a craft position determiner 
for determining the position and heading of the surface-based 
craft using the image and a reconnaissance target identifier 
for identifying targets using the image. 

2 0 In another aspect of the invention, the surface-based 

craft further include instrument suites for collection of 
surface-based information. 

In another aspect of the invention, the surface-based 
craft further include a proximity sensor for detecting an 
25 object in close proximity to the surface-based craft and a 
controller, responsive to the proximity sensor, for avoiding a 
collision with the object. 

BRIEF DESCRIPTION OF THE DRAWINGS 

3 0 These and other features, aspects, and advantages of the 

present invention will become better understood with regard to 
the following description, appended claims, and accompanying 
drawings where: 

FIG. la is a block diagram of a multi-agent autonomous 
35 system in accordance with an exemplary embodiment of the 
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present invention; 

FIG. lb is a block diagram illustrating the use of a 
5 multi-agent autonomous system to explore an area in accordance 

with an exemplary embodiment of the present invention; 

FIG. 2 is a block diagram illustrating communication 

links within a multi-agent autonomous system in accordance 

with an exemplary embodiment of the present invention; 
10 FIG. 3 is a block diagram of an agent or craft in 

accordance with an exemplary embodiment of the present 

invention; 

FIG. 4 is a block diagram of a craft tracking and command 
system in accordance with an exemplary embodiment of the 
15 present invention; 

FIG. 5 is a software module diagram of a multi-agent 
autonomous system in accordance with an exemplary embodiment 
of the present invention; 

FIG. 6 is a process flow diagram for a multi-agent 
20 autonomous system in accordance with an exemplary embodiment 
of the present invention; 

FIG. 7a is a block diagram of a craft imaged in an 
environment by a craft tracking and command system in 
accordance with an exemplary embodiment of the present 
25 invention; 

FIG. 7b is a block diagram of a craft, targets, and 
obstacles identified in an environment by a craft tracking and 
command system in accordance with an exemplary embodiment of 
the present invention; 
30 FIG. 7c is a block diagram of a planned craft path in 

accordance with an exemplary embodiment of the present 
invention; 

FIG. 8 is a block diagram depicting an iterative path 
planning sequence in accordance with an exemplary embodiment 
35 of the present invention; 
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FIG. 9 is a process flow diagram of an image processing 
system in accordance with an exemplary embodiment of the 
5 present invention; 

FIG. 10 is a process flow diagram of a craft command 
process in accordance with an exemplary embodiment of the 
present invention; and 

FIG. 11 is an architecture diagram of a data processing 
10 apparatus suitable for use as a craft tracking and command 
system in accordance with an exemplary embodiment of the 
present invention . 

DETAILED DESCRIPTION 

15 A multi-agent autonomous system includes numerous and 

redundant, specialized, low cost, expendable, sensor-equipped 
self-propelled surface-based reconnaissance craft, that 
collectively can cover vast expanses of terrain in a 
relatively short amount of time. Furthermore, in utilizing an 

20 overhead view, several significant mission advantages emerge, 
such as: enhanced surface-based craft safety (e.g., cliffs are 
visible long before the rover gets there) ; more efficient, 
air-controlled path planning for surface-based craft, thus 
increased mission science return; and enhanced control of 

2 5 surface-based craft traverses in unknown terrain. 

An integrated air-ground multi-agent autonomous remote 
planetary surface exploration allows truly autonomous science 
exploration missions with air and ground-based agents in real 
environments. Furthermore, an overhead perspective allows for 

3 0 unprecedented safety of surface-based craft, i.e., non- 

traversable terrain (e.g., cliffs) is detected long before a 
surface-based craft gets there. Also, the overhead view 
allows for much more efficient path planning simply because 
more terrain is visible. This is particularly important when 
35 surface-based craft leave an operational area imaged during a 
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descent onto a planetary surface. Optimized path planning 
leads to increased mission objective return. The advantage of 

5 having an overhead view reduces drastically the planning 

effort necessary to navigate the surface-based craft and thus 
allows for commanding multiple surface-based craft with almost 
no additional effort. 

The multi-agent autonomous system can be applied to a 

10 variety of missions, such as: mine-sweeping operations; clean- 
up operations in hazardous environments (e.g., chemical 
plants, nuclear plants, etc.); scientific operations such as 
sample detection and sample return (e.g., search for 
meteorites in Antarctica) ; and military reconnaissance 

15 operations in hostile environments. 

FIG. la is a block diagram of a multi-agent autonomous 
system in accordance with an exemplary embodiment of the 
present invention. A multi-agent autonomous system includes 
an air-borne tracking and command system 100a that may be held 

20 aloft by a support platform such as a lighter- than-air 
aircraft or balloon 101a. The tracking and command system is 
in communication with one or more surface-based craft or 
agents, such as craft 102a and 104a. Communications may be 
accomplished by radio control communications links 103a and 

25 105a. 

Each craft includes an instrument suite, such as 
instrument suites 108a and 110a that are used by the craft to 
obtain information about an environment or operational area 
that the craft are deployed in. For example the instrument 

30 suite may include an optical camera for taking images of an 
area immediately around each of the craft. Other imaging 
systems and instruments may be employed as well. The sensor 
signals from the instrument suite are transmitted by the craft 
to the tracking and command system. 

35 The tracking and command system includes its own 
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instrument suite 112a. The tracking and command system uses 
its instrument suite to take sensor readings of the craft and 
5 the environment surrounding the craft. For example, the 

tracking and command system's instrument suite may include an 
operational area imager such as an optical camera for 
capturing images of the craft environment and the craft within 
the environment . 

10 In operation, the tracking and command system receives 

signals from its instrument suite including environment and 
craft signals indicating the position and heading of each 
craft in the environment. The tracking and command system 
uses the signals to generate a set of craft movement commands 

15 that are transmitted to the craft. The craft respond to the 
craft movement command signals by moving in the commanded 
manner through the environment . The tracking and command 
system also generates craft command signals commanding the 
craft to employ their own instrument suites to investigate 

2 0 objects or targets within their environment. 

As the tracking and command system is airborne above the 
craft environment, the tracking and command system has a much 
larger field of view than the craft. As such, the tracking 
and command system may detect various obstacles and targets in 

2 5 the environment that the surface-based craft may not be able 
to detect. By having a larger field of view of the 
environment, the tracking and command system may select 
targets for exploration by the craft that the surface-based 
craft are unable to select simply because the surface-based 

30 craft cannot detect the potential target in the first place. 

In addition, the tracking and command system may use its 
larger field of view to more accurately determine a path for 
each craft around obstacles in the craft ! s environment. 

The aircraft supporting the tracking and command system 

35 may further include a thrust generating element 114a for 
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maneuvering the aircraft. Maneuvering the aircraft may be 
useful for both ensuring the aircraft remains in a desired 

5 area or for moving the multi -agent autonomous system to a new 

operational area. In addition, various forms of aircraft may 
be used as a platform for the tracking and command system. 
For example, the aircraft may be some other type of lighter- 
than-air aircraft such as a blimp. The aircraft may also be a 

10 heavier-than-air aircraft such as a glider, airplane, or 
helicopter. 

In another tracking and command system support platform 
in accordance with an exemplary embodiment of the present 
invention, the platform is a tethered lighter-than-air 

15 aircraft. The aircraft may be tethered either directly to the 
surface, and thus fixed in place, or may be tethered to one of 
the surface-based craft. If tethered to one of the surface- 
based craft, the support platform and tracking and command 
system may travel to different locations along with the 

20 surface-based craft. 

In another embodiment, the deployed agents may include 
one or more non-mobile sensors, such as sensor 109a, that are 
not self-propelled. These additional sensors may be used to 
augment or replace the surface-based information collected by 

25 the mobile surface-based craft. 

FIG. lb is a block diagram illustrating the use of a 
multi-agent autonomous system to explore an area in accordance 
with an exemplary embodiment of the present invention. One or 
more multi-agent autonomous systems, such as multi-agent 

30 autonomous systems 116a and 116b, may be coupled to a 
satellite 118 for exploration of a large area. The satellite 
may include its own instrument suite 119 for imaging the area 
being explored by the multi-agent autonomous systems. 
Information collected by the multi-agent autonomous systems 

35 and the satellite using its instrument suite is integrated 120 
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to generate a database 122 including views of the explored 
area generated by the various components of the exploration 

5 system. For example, information supplied by the surface- 

based craft 124 may include the detailed images of an 
operational area. However, since the surface-based craft have 
only a limited view, the actual area imaged by the surface- 
based craft may be small. Information supplied by the 

10 airborne tracking and command systems 126 may include images 
of a large portion of the explored area. However, as the 
airborne tracking and command systems are more elevated and 
further away from the explored area with respect to the 
surface-based craft, the information supplied by the tracking 

15 and command systems may be less detailed than the information 
collected by the surface-based craft. Information supplied by 
the satellite may include images from the entire explored 
area, but may not be as detailed as the information supplied 
by the tracking and command systems. By combining information 

2 0 from the components of the exploration system, a large area 

may be explored with a high level of detail. 

Portions of the database of information may be 
transmitted 13 0 to the satellite and distributed to the 
coupled multi-agent autonomous systems. In this way, a multi- 
25 agent autonomous system may use information collected by the 
satellite or another autonomous system to aid in selection of 
targets to be explored by the surface-based craft. 

FIG. 2 is a block diagram illustrating communication 
links within an exploration system using a multi-agent 

3 0 autonomous system in accordance with an exemplary embodiment 

of the present invention. A tracking and command system 100a 
receives an initiation signal 200 from a satellite 118. The 
tracking and command system may also receive information 2 01 
about an area to be explored such as images generated by the 
3 5 satellite's instrument suite 119. In response to the 
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initiation signal, the tracking and command system uses its 
own instrument suite 112a to image the area to be explored. 

5 The tracking and command system uses the information received 

from the satellite and its own imaging information to identify 
targets and obstacles in the area to be explored. Once the 
targets and the objects have been identified, the tracking and 
command system generates a path for a craft 104a to follow to 

10 get from the craft's current position to a target while 
avoiding any obstacles. The tracking and command system uses 
the path to generate and transmit craft command signals 2 02 to 
the craft. The craft responds to the command signals until it 
reaches a target. At the target, the craft uses its own 

15 instrument suite 110a to collect surface-based information 
about the target. The craft transmits the information about 
the target 2 04 to the tracking and command system. The 
tracking and command system in turn transmits its own imaging 
information as well as the craft's surface-based target 

20 information 206 to the satellite for integration into a 
database as previously described. 

In addition to collecting information about targets and 
responding to craft commands from the tracking and command 
system, the craft may also use internal control logic to 

25 internally manage (208) collision avoidance. For example, the 
craft may include proximity sensors used to detect objects or 
other craft in its immediate vicinity. By using signals 
generated from the proximity sensors, the craft may avoid 
collisions with objects that the tracking and command system 

30 may not be able to detect. In a similar manner, the tracking 
and command system may generate (210) its own navigational 
commands in order to travel to a new area for exploration or 
maintain its position within an area of exploration. 

As depicted, the multi-agent autonomous system is a 

35 multilayered and hierarchal system. For example, the surface- 
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based craft constitute one layer, the tracking and command 
system constitute another layer, and the satellite constitutes 

5 yet another layer. Each layer provides both information 

inputs from specific instrument suites and also include 
computational elements. The exact distribution of the 
instrument suites, computational elements, and even the number 
of layers may be altered. For example, an extra airborne 

10 layer may be added to command the tracking and command system 
to relocate. In addition, the computations performed by the 
tracking and command system may be performed by a surface- 
based craft or base station. Finally, the satellite may be 
used to track and command the surface-based craft without 

15 having an intervening airborne instrumented or processing 
element . 

FIG. 3 is a block diagram of a surface-based agent or 
craft in accordance with an exemplary embodiment of the 
present invention. A craft 102 includes a controller 300 

2 0 having programming instructions 3 01 for controlling the 

operation of the craft. The controller is coupled to a 
transceiver 302 and antenna 304 for receiving and transmitting 
signals 305 from a tracking and command system. The 
controller is further coupled to an instrument suite 204 used 
25 to analyze the craft's environment. The sensor suite may 
include imaging sensors such as video cameras for capturing 
images of the environment for transmission to the tracking and 
command system. 

The instrument suite may further include proximity 

3 0 sensors used to sense objects or other craft in the immediate 

vicinity of the craft. The crafts controller may be 
programmed to use signals received from the proximity sensors 
to avoid collisions with obstacles or other craft. 

The controller is further coupled to a drive controller 
3 5 3 06 used to operate the craft's drive mechanism 3 08. In one 
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surface-based craft in accordance with an exemplary embodiment 
of the present invention, the surface-based craft includes a 

5 tread drive mechanism. In other surface-based craft, the 

drive mechanism may include wheels, legs operable to move the 
craft, surface effect drives, etc. In addition, the surface- 
based craft may be operable on the surface of a body of water. 
Such a craft may be amphibious or be a boat or ship. 

10 In one multi-agent autonomous system in accordance with 

an exemplary embodiment of the present invention, an 
individually addressable Radio Controlled (R/C) robot unit is 
used as a surface-based craft. The robot unit is supplied by 
Plantraco Ltd. of Saskatoon, Canada, and is known as a 

15 "Telecoramander Desktop Sciencecraf t " . 

Each Telecommander Desktop Sciencecraft system includes a 
sciencecraf t unit and a Universal Serial Bus (USB) -controlled 
R/C commanding unit. A tracking and command system issues 
commands to the deployed sciencecraft via the USB-connected 

20 R/C control unit. Each sciencecraft operates on a unique R/C 
frequency. 

In addition, the Telecommander Desktop Sciencecraft 
contains an integrated onboard color video camera. When the 
sciencecraft arrives at a destination, it can relay in-situ 

25 images of the target back to the tracking and command system 
for science analysis. 

FIG. 4 is a block diagram of a craft tracking and command 
system in accordance with an exemplary embodiment of the 
present invention. A tracking and command system 100 includes 

30 a controller 400 for controlling the operations of the 
tracking and command system. The controller is coupled to a 
satellite transceiver 402 for communicating with a satellite 
118. The controller is further coupled to a surface-based 
craft transceiver 404 used to communicate with a surface based 

35 craft 102. The controller is further coupled to an instrument 
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suite interface 406. The controller uses the instrument suite 
interface to control the operations of an instrument drive 408 

5 mechanically coupled to an instrument suite 200. The 

instrument drive may be used to aim, focus, and adjust the 
magnification of imaging sensors such as optical cameras. The 
instrument suite is electrically coupled to the instrument 
suite interface for use by the controller in collection of 

10 information used by the tracking and command system to track 
the surface-based craft and generate paths for the surface- 
based craft. 

If the tracking and command system is mounted on a 
platform that is capable of transporting the tracking and 

15 command system between operational areas, the controller is 
further coupled to a platform drive interface 410. The 
platform drive interface is further coupled to a platform 
drive mechanism 412. 

FIG. 5 is a software module diagram of a multi-agent 

20 autonomous system in accordance with an exemplary embodiment 
of the present invention. The tracking and command system 
includes a communications module 500 for receiving commands 
from a satellite or other external system. Once an initiate 
signal is received, the tracking and command system uses an 

25 overhead image capturing module 501 to capture an image of an 
operational area. The image 502 is transmitted to a 

reconnaissance craft position module 504. The reconnaissance 
craft position module uses the image to determine the location 
and heading 506 of a surface-based craft in the area of 

30 operation. The image is further used by an imaging processing 
module 512 to identify any obstacles that may be in the area 
to be explored by generating a set of obstacle coordinates and 
outlines or edges 514. A processed image 516 from the image 
processing module is transmitted to a feature detection module 

35 518. The feature detection module uses the processed image to 
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demark image features 520 that are transmitted to a 
reconnaissance and target identification module 522. The 

5 reconnaissance and target identification identifies features 

of interest in the operational area. Those features of 
highest interest are identified as targets and a set of target 
positions 523 is generated. 

A path planning module 508 receives the reconnaissance 

10 craft location information 506, the obstacle location and edge 
information 514, and target positions 523. The path planning 
module uses this information to plan a path for the generate a 
set of craft commands 524. The tracking and command system 
then uses a craft communication module 52 6 to transmit the 

15 craft commands to a craft. 

The craft communication module is also used to receive 
transmissions from a craft. The transmissions may include 
information collected by the craft using the craft's own 
instrumentation suite, such as a camera image 528. This 

2 0 information is provided to the path planning module along with 
the information taken from the tracking and command system. 
The path planning module may use the craft instrument suite 
information to further refine the craft commands to be sent to 
the craft. 

2 5 The software modules continuously process information 

received from the tracking and command system instrument suite 
and the craft instrument suite as indicated by feedback loop 
526. This generates a constant stream of craft commands that 
are transmitted to the surface-based craft as they travel 

3 0 within the operational area. 

The path planning forwards the craft's instrument suite 
information 530 to an in-situ measurement module 532. The in- 
situ measurement module analyzes the information received from 
the surface-based craft and forwards the resultant 
35 measurements 533 to an intelligence reconnaissance output 
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module 534. The intelligence reconnaissance output module 
transmits the craft information to external entities for 
5 integration into a previously described database 122 (of FIG. 

lb) . 

FIG. 6 is a process flow diagram for a multi-agent 
autonomous system in accordance with an exemplary embodiment 
of the present invention. The process flow diagram 

10 illustrates the processing sequence of the software modules of 
FIG. 5. A tracking and command system receives an exploration 
initiation signal 600 and iterates (602) the following 
sequence of operations. In a science and imaging process 
phase 604, the tracking and command system collects images 

15 about an operational area including surface-based craft 
deployed in the operational area. In a craft path 

determination phase (606) the tracking and command system 
determines a pathway to be followed by a craft in the 
operational area. In a craft command phase (608) , the 

2 0 tracking and command system commands a craft 102 in such a way 
as the craft moves along a designated path between obstacles 
and towards a target. The process repeats iteratively 602 
such that the craft follows a pathway to a target location. 

In slightly more detail, the science and imaging process 

2 5 phase further includes receiving an image 609 of an 

operational area including images of any craft deployed in the 
operational area. The tracking and command system determines 
(610) from the image the location and heading of any craft in 
the operational area. The tracking and command system also 

3 0 acquires (612) the image and processes (614) the image to 

determine (616) obstacles and targets within the operational 
area. 

During the craft path determination phase, the tracking 
and command system uses the obstacles, targets, and craft 
35 current position to determine (618) a navigation baseline for 
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each craft deployed in the operational area. The tracking and 
command system then generates (620) a pathway for each craft 

5 to follow so that the craft may avoid the identified obstacles 

and reach a target . 

FIG. 7a is a semi -schematic drawing of a craft imaged in 
an environment by a craft tracking and command system in 
accordance with an exemplary embodiment of the present 

10 invention. The image 700 includes an image taken of a craft 
702. The craft includes markings or other indica that are 
visible to the tracking and command system's instrument suite. 
The markings may be used by the tracking and command system to 
determine a craft's heading. The image further includes 

15 images of obstacles,, such as obstacles 704a, 704b, and 704c, 
that may impede the progress of the craft as the craft moves 
around within the imaged area. The image further includes 
images of targets, such as 710a and 710b, that the tracking 
and command system may determine are of interest. The 

20 tracking and command system will use the image to determine a 
path for the craft to take to reach the targets while avoiding 
the obstacles. 

FIG. 7b is a semi-schematic drawing of a craft, targets, 
and obstacles identified in an environment by a craft tracking 

25 and command system in accordance with an exemplary embodiment 
of the present invention. The tracking and command system 
identifies the craft 702 in the image by extracting features 
from the image and analyzing the features. Once the tracking 
and command system has identified the craft, as exemplified by 

30 the box 712a, the tracking and command system can determine 
the craft position. In addition, the tracking and command 
system uses the indica on the craft to determine the heading 
of the craft. 

The tracking and command system also separates other 
35 features within the image into obstacles and targets. A 
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target may be separated from an obstacle by considering a 
feature space including the objects size, shape, albedo, 

5 surface irregularities, etc. Once an object has been 

identified as a target, as exemplified by box 712b around 
target 710a, the tracking and command system can determine the 
targets position. The tracking and command system continues 
processing features in the image to identify other targets, 

10 such as target 710b, as exemplified by triangle 712c. Other 
features identified by the tracking system, such as features 
704a, 704b, and 704c, are identified as obstacles to be 
avoided by the craft. 

FIG. 7c is a semi -schematic drawing of a planned craft 

15 path in accordance with an exemplary embodiment of the present 
invention. The craft is commanded by the tracking and command 
system to travel through the imaged operational area through a 
sequence of moves. For example, the craft 7 02 may be 
commanded to rotate a specified number of degrees in order to 

2 0 adjust its heading so that the craft is pointed toward its 
next target 710a. The craft is then commanded to travel a 
certain distance along its path, thus defining a segment 714b 
of a path. The path segment avoids obstacles 704a and 704b 
and takes the craft to its first target. When the craft 

2 5 arrives at the first target, the craft is then commanded to 
rotate and travel along path segment 714a to a second target 
710b, while avoiding obstacle 704c. In this way, the craft is 
commanded to travel throughout an operational area, traveling 
from target to target without becoming obstructed by an 

30 obstacle. As the tracking and command system has information 
about a large area, it may intelligently command a craft 
through the large area without placing the craft into an 
untenable position. 

FIG. 8 is a semi-schematic drawing depicting an iterative 

35 path planning sequence in accordance with an exemplary 
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embodiment of the present invention. A craft 702 may be 
commanded to pass through an area to target 710b. As the 

5 tracking and command system may take a sequence of images in 

order to command the craft, the tracking and command system 
may issue a sequence of craft commands adjusting the craft's 
path through the environment in incremental steps. In this 
way, the multi-agent autonomous system may correct any errors 

10 that occur in the craft's progress. For example, the tracking 
and command system may command the craft to travel along path 
segment 800a. The tracking and command system issues a 
rotation command and a move forward command to the craft. As 
the craft moves forward, it may deviate from the desired path 

15 as indicated by segment 800b, In subsequent iterative steps, 
the tracking and command system takes images of the 
operational area and calculates additional segments for the 
path. For example, the tracking and command system may 
command the craft to rotate and follow path segment 802a only 

2 0 to discover that the craft has traveled along path segment 
802b. Upon each iteration, the tracking and command system 
may issue commands guiding the craft along successive 
segments, such as 800a, 802a, and 804a, each time correcting 
the path of the craft when the tracking and command system 

2 5 determines that the craft has actually traveled along other 

path segments, such as 800b, 802b, and 804b. Thus, through 
successive craft commands, the tracking and command system 
guides the craft along successive path segments, resulting in 
the craft arriving at the desired target 710b. 

3 0 FIG. 9 is a process flow diagram of an image processing 

system in accordance with an exemplary embodiment of the 
present invention. A tracking and command system processes 
images from its instrument suite to determine a path for a 
craft. The tracking and command system does so by receiving 
3 5 an image and extracting information from the image in a 



-21- 



1 50540/FLC/C766 - CIT3728 

sequence of passes, saving intermediate results in a temporary 
datastore. In a first pass, the tracking and command system 

5 converts (900) an input file image 902 into an image file 903 

including color information and an image file 904 wherein the 
colors have been mapped to a grayscale. The tracking and 
command system then extracts (906) feature information 908 
from the grayscale image file. The tracking and command 

10 system uses the feature information to identify (910) 
obstacles and targets and extract locations 912 of the 
obstacles and targets. The tracking and command system then 
uses the image file and the locations to generate (914) a mark 
feature file 916 of features that the tracking and command 

15 system may find to be interesting . 

The tracking and command system next uses the image file 
and the location file to generate (918) a color file 920 
wherein the color of each object, such as obstacles or 
targets, is indicated. The tracking and command system uses 

20 the image file and the location file to generate (922) an 
albedo file wherein the albedo of each object, such as 
obstacles or targets, is indicated. 

The image file, color file, and albedo file are used by 
the tracking and command system to generate (924) a target 

25 image file 932 and a target file 934. The location file is 
finally used to generate (936) a navigation file 938 and a 
science target file 940 for use in determining a path for a 
craft . 

Algorithms for extracting features from an image, 
30 including extracting the position and heading of a man-made 
object in a natural environment, are well known in the art of 
robotics. Each algorithm has its own advantages and 

weaknesses. As such, multi-agent autonomous systems in 
accordance with exemplary embodiments of the present invention 
35 may include different feature extraction algorithms dependent 
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on the needs of a researcher or explorer. In one multi-agent 
autonomous system in accordance with an exemplary embodiment 

5 of the present invention, operational area analysis is 

performed by a suite of imaging processing programs entitled 
"Automated Geologic Field Analyzer" developed at the Jet 
Propulsion Laboratory of Pasadena, California. In addition, 
craft tracking is performed using a neural network using 

10 adaptive Radial Basis Functions (RBFs) for target recognition 
and tracking as described in "Real-time automatic target 
recognition using a compact 512 x 512 grayscale optical 
correlator, T. Chao, H. Zhou, G. F. Reyes, J. Hanan; 
Proceedings of SPIE Vol. #5106, the contents of which are 

15 hereby incorporated by reference as if stated fully herein. 

FIG. 10 is a process flow diagram of a craft command 
process in accordance with an exemplary embodiment of the 
present invention. A tracking and command system uses a craft 
command process to generate paths for craft deployed in an 

20 operational area. From the paths, specific craft commands are 
generated. To generate a path, a navigation baseline process 
618 receives a navigation file 938 and science goal file 940 
generated in the previously described image processing process 
of FIG. 9. The navigation baseline process generates a 

25 landscape map 1000 and a navigation map 1002. A pathway 
generation process 620 uses the landscape map, the navigation 
map, and the target file 934 to generate a pathway map 1004. 
From the pathway map, individual craft commands are generated 
that are transmitted to a craft 102 in a craft commanding 

30 process 610. 

Algorithms for determining paths for a robot in an 
accurately characterized environment are well known in the art 
of robotics. Each algorithm has its own advantages and 
weaknesses. As such, multi-agent autonomous systems in 

35 accordance with exemplary embodiments of the present invention 
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may include different path finding algorithms dependent on the 
needs of a researcher or explorer. In one multi-agent 

5 autonomous system in accordance with an exemplary embodiment 

of the present invention, path finding is performed using a 
line intersection method. Other algorithms may include 
weighted graph algorithms, the well-known A* method, etc. 

FIG. 11 is an architecture diagram of a data processing 

10 apparatus suitable for use as a craft tracking and command 
system controller in accordance with an exemplary embodiment 
of the present invention. The data processing apparatus 400 
includes a processor 1100 coupled to a main memory 1102 via a 
system bus 1104. The processor is also coupled to a data 

15 storage device 1106 via the system bus. The storage device 
includes programming instructions 1108 implementing the 
features of a tracking and command system as described above. 
In operation, the processor loads the programming instructions 
into the main memory and executes the programming instructions 

2 0 to implement the features of the tracking and command system. 

The data processing system may further include a 
plurality of communications device interfaces 1110 coupled to 
the processor via the system bus. A tracking and command 
system controller, hosted by the data processing system, uses 

2 5 the communications device interfaces to communicate with 

surface-bound craft or satellite as previously described. 

The data processing system may further include an 
instrument interface 1114 coupled to the processor via the 
system bus. A tracking and command system controller, hosted 

3 0 by the data processing system, uses the instrument interface 

to generate control signals for a tracking and imaging 
instrument suite as previously described. In addition, the 
instrument interface is used by the tracking and command 
system controller to receive instrument suite sensor signals 
35 such as images of the operational area. 
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The data processing system may further include a platform 
drive interface 1116 coupled to the processor via the system 

5 bus. A tracking and command system controller, hosted by the 

data processing system, uses the platform drive interface to 
generate control signals for a platform supporting the 
tracking and command system. 

Although this invention has been described in certain 

10 specific embodiments, many additional modifications and 
variations would be apparent to those skilled in the art. It 
is therefore to be understood that this invention may be 
practiced otherwise than as specifically described. Thus, the 
present embodiments of the invention should be considered in 

15 all respects as illustrative and not restrictive, the scope of 
the invention to be determined by any claims supported by this 
application and the claims' equivalents rather than the 
foregoing description. 
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